In this paper, we propose a spatial propagation networks for learning affinity matrix. We show that by constructing a row/column linear propagation model, the spatially variant transformation matrix constitutes an affinity matrix that models dense, global pairwise similarities of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that is effective to model any task-specific pairwise similarity.

The authors incorporate ideas from image processing into CNNs and show how nonlinear diffusion can be combined with deep learning. This allows to train more accurate post-processing modules for semantic segmentation, that are shown to outperform denseCRF-based post-processing, or recurrent alternatives that rely on more straightforward interpretations of recursive signal filtering, as introduced in [3,16]. The main practical contribution lies in extending the techniques of [3,16]: when these techniques apply recursive filtering, say in the horizontal direction of an image, they pass information along rows in isolation. Instead the method of the authors allows one to propagate information across rows, by rephrasing the originally scalar recursion in terms of vector-matrix products. This is shown to be much more effective than the baseline.

In this paper, we propose a spatial propagation networks for learning affinity matrix. We show that by constructing a row/column linear propagation model, the spatially variant transformation matrix constitutes an affinity matrix that models dense, global pairwise similarities of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that is effective to model any task-specific pairwise similarity. The spatial propagation network is a generic framework that can be applied to numerous tasks, which traditionally benefit from designed affinity, e.g., image matting, colorization, and guided filtering, to name a few. Furthermore, the model can also learn semantic-aware affinity for high-level vision tasks due to the learning capability of the deep model.

AI has become part of the public consciousness. Researchers and data scientists have been sharing their groundbreaking work -- at what is officially known as the Conference and Workshop on Neural Information Processing Systems -- for three decades. But it's only with the recent explosion of interest in deep learning that NIPS has really taken off. We had two papers accepted to the conference this year, and contributed to two others. The researchers involved are among the 120 people on the NVIDIA Research team focused on pushing the boundaries of technology in machine learning, computer vision, self-driving cars, robotics, graphics, computer architecture, programming system, and other areas.

In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be outputs from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, such as image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically aware affinity values for high-level vision tasks due to the powerful learning capability of deep CNNs.